[SPARK-7411] [SQL] Support SerDe for HiveQl in CTAS #5963

chenghao-intel · 2015-05-07T04:26:10Z

This is a follow up of #5876 and should be merged after #5876.

Let's wait for unit testing result from Jenkins.

AmplabJenkins · 2015-05-07T04:27:10Z

Merged build triggered.

AmplabJenkins · 2015-05-07T04:27:16Z

Merged build started.

SparkQA · 2015-05-07T04:28:57Z

Test build #32072 has started for PR 5963 at commit a8575a0.

SparkQA · 2015-05-07T04:30:27Z

Test build #32072 has finished for PR 5963 at commit a8575a0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CreateTableAsSelect(

AmplabJenkins · 2015-05-07T04:30:28Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T04:30:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32072/
Test FAILed.

AmplabJenkins · 2015-05-07T04:52:11Z

Merged build triggered.

AmplabJenkins · 2015-05-07T04:52:19Z

Merged build started.

SparkQA · 2015-05-07T04:54:00Z

Test build #32077 has started for PR 5963 at commit 889d822.

SparkQA · 2015-05-07T07:08:15Z

Test build #32077 has finished for PR 5963 at commit 889d822.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CreateTableAsSelect(

AmplabJenkins · 2015-05-07T07:08:20Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T07:08:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32077/
Test PASSed.

yhuai · 2015-05-08T02:39:23Z

@chenghao-intel I have merged #5876.

AmplabJenkins · 2015-05-09T14:22:10Z

Merged build triggered.

AmplabJenkins · 2015-05-09T14:22:19Z

Merged build started.

SparkQA · 2015-05-09T14:24:03Z

Test build #32302 has started for PR 5963 at commit f4e243f.

SparkQA · 2015-05-09T16:15:08Z

Test build #32302 has finished for PR 5963 at commit f4e243f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-09T16:15:12Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-09T16:15:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32302/
Test PASSed.

marmbrus · 2015-05-09T20:48:57Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala

+          // parquet.hive.DeprecatedParquetInputFormat                     => Parquet
+          // TODO configurable?
+          format.contains("Orc") || format.contains("Parquet") || format.contains("RCFile")
+        }).getOrElse(false))


I'm not sure this is really the logic we want. The goal here is to by default (i.e. if the user does not specify anything about storage), when convertCTAS is turned on to use the data sources API. Would it be possible to have the parser only fill in the storage options when the user specifies them and defer filling in default values until we are in the analyzer. That way we can distinguish "no storage options specified" from "default storage options chosen".

Ok, I see, I will move the default SerDe from HiveQl to Analyzer

marmbrus · 2015-05-09T20:53:43Z

This is looking pretty good. Thanks for taking the time to flesh this part out.

AmplabJenkins · 2015-05-10T15:07:11Z

Merged build triggered.

AmplabJenkins · 2015-05-10T15:07:16Z

Merged build started.

SparkQA · 2015-05-10T15:09:01Z

Test build #32343 has started for PR 5963 at commit a8260e8.

AmplabJenkins · 2015-05-10T15:22:11Z

Merged build triggered.

AmplabJenkins · 2015-05-10T15:22:21Z

Merged build started.

SparkQA · 2015-05-10T15:24:04Z

Test build #32344 has started for PR 5963 at commit f87ace6.

SparkQA · 2015-05-10T16:59:34Z

Test build #32343 has finished for PR 5963 at commit a8260e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-10T16:59:38Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-10T16:59:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32343/
Test PASSed.

SparkQA · 2015-05-10T17:54:05Z

Test build #32344 timed out for PR 5963 at commit f87ace6 after a configured wait of 150m.

AmplabJenkins · 2015-05-10T17:54:10Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-10T17:54:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32344/
Test FAILed.

chenghao-intel · 2015-05-11T00:14:32Z

retest this please.

AmplabJenkins · 2015-05-11T00:17:11Z

Merged build triggered.

AmplabJenkins · 2015-05-11T00:17:18Z

Merged build started.

SparkQA · 2015-05-11T00:18:57Z

Test build #32355 has started for PR 5963 at commit f87ace6.

SparkQA · 2015-05-11T02:08:00Z

Test build #32355 has finished for PR 5963 at commit f87ace6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-11T02:08:04Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-11T02:08:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32355/
Test PASSed.

chenghao-intel · 2015-05-11T02:27:01Z

@marmbrus @yhuai Any more comments?

This is a follow up of #5876 and should be merged after #5876. Let's wait for unit testing result from Jenkins. Author: Cheng Hao <[email protected]> Closes #5963 from chenghao-intel/useIsolatedClient and squashes the following commits: f87ace6 [Cheng Hao] remove the TODO and add `resolved condition` for HiveTable a8260e8 [Cheng Hao] Update code as feedback f4e243f [Cheng Hao] remove the serde setting for SequenceFile d166afa [Cheng Hao] style issue d25a4aa [Cheng Hao] Add SerDe support for CTAS (cherry picked from commit e35d878) Signed-off-by: Michael Armbrust <[email protected]>

This is a follow up of apache#5876 and should be merged after apache#5876. Let's wait for unit testing result from Jenkins. Author: Cheng Hao <[email protected]> Closes apache#5963 from chenghao-intel/useIsolatedClient and squashes the following commits: f87ace6 [Cheng Hao] remove the TODO and add `resolved condition` for HiveTable a8260e8 [Cheng Hao] Update code as feedback f4e243f [Cheng Hao] remove the serde setting for SequenceFile d166afa [Cheng Hao] style issue d25a4aa [Cheng Hao] Add SerDe support for CTAS

chenghao-intel added 3 commits May 9, 2015 21:28

Add SerDe support for CTAS

d25a4aa

style issue

d166afa

remove the serde setting for SequenceFile

f4e243f

chenghao-intel force-pushed the useIsolatedClient branch from 889d822 to f4e243f Compare May 9, 2015 14:22

marmbrus reviewed May 9, 2015
View reviewed changes

Update code as feedback

a8260e8

remove the TODO and add resolved condition for HiveTable

f87ace6

chenghao-intel changed the title ~~[SPARK-7411] [SQL] [WIP]Support SerDe for HiveQl in CTAS~~ [SPARK-7411] [SQL] Support SerDe for HiveQl in CTAS May 10, 2015

asfgit closed this in e35d878 May 12, 2015

[SPARK-7411] [SQL] Support SerDe for HiveQl in CTAS #5963

[SPARK-7411] [SQL] Support SerDe for HiveQl in CTAS #5963

Conversation

chenghao-intel commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

yhuai commented May 8, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

marmbrus May 9, 2015

Choose a reason for hiding this comment

chenghao-intel May 10, 2015

Choose a reason for hiding this comment

marmbrus commented May 9, 2015

AmplabJenkins commented May 10, 2015

AmplabJenkins commented May 10, 2015

SparkQA commented May 10, 2015

AmplabJenkins commented May 10, 2015

AmplabJenkins commented May 10, 2015

SparkQA commented May 10, 2015

SparkQA commented May 10, 2015

AmplabJenkins commented May 10, 2015

AmplabJenkins commented May 10, 2015

SparkQA commented May 10, 2015

AmplabJenkins commented May 10, 2015

AmplabJenkins commented May 10, 2015

chenghao-intel commented May 11, 2015

AmplabJenkins commented May 11, 2015

AmplabJenkins commented May 11, 2015

SparkQA commented May 11, 2015

SparkQA commented May 11, 2015

AmplabJenkins commented May 11, 2015

AmplabJenkins commented May 11, 2015

chenghao-intel commented May 11, 2015